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AESTRACT 



Scott's coefficient is an improved method for 
estimating observer reliability using any system which assigns events 
to mutually exclusive categories. The formula is the ratio of the 
actual difference between obtained and chance agreement to the 
maximum difference between obtained and chance agreement. A FORTRAN 
program for calculating this reliability coefficient is presented. 

The program can be used with from 2 to 20 observers and can include 
any number of observations as long as the product of the observations 
times the observers does not exceed 200. The observation system 
cannot exceed 30 categories. There are four outputs of the program 
including the average reliability coefficient of all possible pairs 
of observers, the average difference between all possible pairs cf 
observers for each category, the percentage of each observer's total 
assignments fcund in each category, and each observer's average 
coefficient cf reliability with all other observers. The data deck 
arrangement ard program listing are included. (PR) 
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A COMPUTER PROGRAM FOR CALCULATION OF SCOTT *S 
COEFFICIENT OF OBSERVER RELIABILITY 



By 



Thomas B> Gregory 1 
Indiana University 



The past decade has witnessed the development of numerous 
observation systems that attempt to objectively record the 
ongoing verbal and nonverbal events occurring in the classroom. 
F lander * s Interaction Analysis (1960) and the several revisions 
of the OScAR (e.g., Medley and Mitael, 1953; Medley and others, 
1968) are notable examples of this trend. Since such descrip- 
tive systems increasingly are being utilised as criterion 
measures in educational research, the problem of obtaining 
valid estimates of inter -observer reliability is a crucial 
one . 

Scott (1955) proposed a method of estimating the reli- 
ability of observers using any observation system that assigns 
events to mutually exclusive categories . Tne method marks an 



^Formerly, Department of Curriculum and Instruction, College 
of Education, The University of Texas at Austin. 
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improvement over conventional methods that are based on the 
percentage of observer agreements P Since such methods re- 
quire the assumption of equi-probability of occurrence for 
all categories, they are inappropriate for these observation 
systems, The formula for Scott 1 s coefficient is: 

If - P o • P e 

l * P 

e 

where P q (observed percent of agreement) represents the per- 
centage of judgements on which two observers agree when 
coding the same data independently; avid ? is the percent 
of agreement to be expected by chance. 'Jf , then, is the 
ratio of the actual difference between obtained and chance 
agreement to the maximum difference between obtained and chance 
agreement (Scott, 1955, p. 323) • Detailed procedures for 
hand-calculation of 'Jf are found elsewhere (Flanders, 1960, 
1967) * 

General Description of thg Computer Program 

This Fortran program may be used with from 2 to 20 ob- 
servers * Any number of individual observations may be included 
as long as the product of the observation N x the observer N 
does not exceed 200 (i.e., the maximum observation N for 20 
observers is 10) „ The observation system cannot exceed 30 
categories = Data may be input as decimal fractions or per- 
cents of the total assignments by each observer or as raw 
scores for each category for each observer. Any input format 
may be specified. Subroutine CCDS (Veldman, 1967, p, 132) 
allows several separate jobs to be "stacked" if the need arises-, 
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Four outputs of the program are: 

1. The average difference between all possible pairs 
of observers for each category. 

2. The percentage of each observer* s total assignments 
found in each category. 

3. The average reliability coefficient of all possible 
pairs of observers. 

4. Each observer* s average coefficient of reliability 
with all other observers. 



Data Deck Arrangement 
Card 1, Title Control Card 
Columns : 

1-80 Title identifying this particular job. It may be 
of any length but cannot be left blank. 



Card 2 Parameter Control Card 
Columns : 



1-5 

6-10 

11-15 

16-20 



21-80 



Number 
Number 
Number 
A 1 if 
a 2 if 
a 3 if 
Blank. 



of observers (max., 20). 

of observations (max., 200 / observer N) . 

of categories in the system (max., 30). 

data is input as percents of total, 

data is input as decimal fractions of total, or 

data is input as raw scores for each category. 



Card 3_ 
Columns : 



1-80 



An appropriate format statement (F mode) for reading 
the data as it is arranged on the tab cards (e.g.. 



(10X, 10F3) ). 



4 



Card 4 Beginning of the data deck arranged with all of 
observer l's observations first, then all of ob- 
server 2's,..»then all of observer N's* 

Last Card Another Title Control Card indicating the beginning 
of a new job or a blank card. 

Program Listing 
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PROGRAM SCOTTR (INPUT* OUTPUT I 

ODIMENSION R( 30*200) * RT(30*20). PI(20*20)* S(200). DIF(30)» 
1 AVG ( 30) * ADIFI30). RAVG(20). KF(16) 

5 CAUL CCDS (KF. NR* NT* NC* NP. N) 

TPI * 0 

DO 15 I = l.NC 
DO 10 J = 1 *NR 
10 RT( I . J) = 0 
15 AD IF( !J = 0 
DO 20 I = l.NR 
20 RAVG( I) = 0 
NRT = NR * NT 

READ KF* ( ( R ( I * J ) . I = 1*NC). J = 1 *NRT ) 

DO 25 I = l.NC 
DO 25 J = 1 .NRT 
25 R(I.J) = R(I.J) + 0.00001 
GO TO (55. 45. 30) NP 
30 DO 35 I = l.NC 
DO 35 J = 1 .NRT 
35 S(J) = SUMF (R. J. NC. 30) 

DO 40 J = l.NRT 
DO 40 I = l.NC 
40 R(I.J) a R(I.J) / S(J) 

45 DO 50 J = l.NRT 
DO 50 I = l.NC 
50 Ri'I.J) = R(I.J) » 100 
55 NT1 = 1 
NT2 =» NT 
DO 70 J = l.NR 
DO 65 I = l.NC 
DO 60 K = NT1 . NT2 
60 RT(I.J) = RT(I.J) + R(I.K) 

65 RT(I.J) = RT ( I . J ) / NT 
NT1 a NT1 + NT 
70 NT2 = NT2 ♦ NT 
DO 85 I = l.NR 
DO 85 J * l.NR 
IF (I .NE. J) GO TO 75 
PI(I.J) = 1 
SO TO 85 

75 DO 80 K = l.NC 

DIF(K) = ABS (RT(K.I) - RT(K.J)) 

ADIF(K) = ADIF(K) + DIF(K> 

80 AVG(K) = (((RT(K.I) + RT(K.J)) / 2) ** 2) * .01 
DIFS = SUMF (DIF. 1. NC. 30) 

AVGS a SUMF (AVG. 1. NC. 30) 

PI(I.J) = (100 - DIFS - AVGS) / (100 - AVGS) 

85 TPI = TPI ♦ PKI.J) 

00-90 K_a_.lj.NC 

90 ADIFIK) = ADIF(K) / (NR * NR - NR) 



TPI 
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(TPI 


- NR) 
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(NR * NR 


DO 


100 


I = 


1*NR 






DO 


95 


J = 


1 * NR 






IF 


(I 


• EO. 


J) GO 


TO 


95 




n n n n n 



RAVG(I) = RAVGII1 + PI(I*J) 

95 CONTINUE 

100 RAVGII) = RAVGUI / (NR - 1> 

PRINT 105 

1050F0RMAT (37H1PERCENT OF TOTAL FOR EACH CATEGORY. // 22H CAT. A. OIF 
1. RATER /) 

PRINT 110* (J* J = l.NR) 

110 FORMAT ( 13X* 2015 /) 

DO 120 I = 1 »NC 

PRINT 115. I* AD I F ( I ) * (RTII.J). J = l.NR) 

115 FORMAT (/ IX. 12* F8.2. 3X. 20F5.1) 

120 CONTINUE 

PRINT 125. TPI 

125 FORMAT (// 39HOCOEFFICIENT FOR ENTIRE RATER GROUP IS . F6.4) 

PRINT 130. NR 

1300FORMAT (/ 66H0SC0TTS RELIABILITY COEFFICIENTS FOR ALL POSSIBLE COM 
13 1 NAT IONS OF . 12. 9H RATERS. /) 

CALL PRTS (PI. NR. NR. 1H . 20) 

PRINT 135 

135 FORMAT (// 34H0AVERAGE R WITH ALL OTHER RATERS. //) 

PRINT 140. (I. RAVGII). I = l.NR) 

140 FORMAT (13. F10.4 /) 

GO TO 5 
END 



SUBROUTINE CCDS (KF* K|» KJ. KK* KL. KM) 

DIMENSION KF( 16) » KH(16) 

READ 5. KH 
5 FORMAT (16A5) 

tF ( KH ( 1 ) .EQ. KH ( 2 ) ) STOP 
READ 10. KI. KJ. KK. KL. KM. KF 
10 FORMAT (515 / 16A5) 

PRINT 15. KH. KI. KJ. KK. KL. KM. KF 
150FORMAT (1H1. 16A5 // 11H PARAMETERS / 13H COL 1- 5 = . 15 / 
113H COL 6-10 = . 15 / 13H COL 11-15 = . 15 / 13H COL 16-20 = . 
215 / 13H COL 21-25 = . 15 // 15H DATA FORMAT = . 16A5) 

RETURN 

END 



SUBROUTINE PRTS (X. N. M. KH. ND) 

DIMENSION X(ND.M) 

IF (M .GT. 1) GO TO 20 
PRINT 15 

DO 10 I = 1* N. 10 
J = MINO (I + 9. N) 

PRINT 5. KH. IS* K = I.J) 

5 FORMAT i/ A7. 10111) 

10 PRINT 15. (X(K.l). K * I.J) 

15 FORMAT (10X. 10fll.4) 

RETURN 

20 DO 25 K » 




1. M. 10 



PRINT 15 

L = MINO ( K + 9. M) 

PRINT 5* KH* (J* J = K.L) 

DO 25 I = 1 * N 

25 PRINT 30* I* (X(I*J)» J = K.L» 
30 FORMAT (/ 16* 4X* 10F11.4) 
RETURN 
END 
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FUNCTION) SUWF ( X « 

X < NO » l ) 



KK «, NN . NO > 



X<K» I ) ** 2 



D ! mfnjS j ON 
SUMF a' O 
N = 1 A6S (NN) 

K = IASS (KKi 

IF ( MM ) 55 * 10 

5 IF IKK1 IB. US* ?5 
10 IF (KX) 35. 55* 45 
1-5 DO 20 I a l*N 
20 SUMF » SUMF 
RETURN 

?5 DO 30 1=1 
30 SUMF a SUMF 4- X<I*K> *•* 
RETURN 

35 DO AO 1=1 *N 
40 SUMF a SUMF 4- 
RETURN 

45 DO 50 J a I *N 
50 SUMF a SUMF + 

«jg RETURN 
CND 



X < K * I > 



X < I iK) 



